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Overview 



Community colleges across the United States face a difficult challenge. On the one hand, they are 
“open access” institutions, with a mission to serve students from all backgrounds and at varying 
levels of college readiness. On the other hand, they must uphold high academic standards in order to 
maintain accreditation and prepare students for employment or transfer to four-year schools. How, 
then, can community colleges best serve students who want to learn but do not meet minimum 
academic standards? 

Chaffey College, a large community college located about 40 miles east of Los Angeles, began to 
wrestle with this question early in the twenty-first century. Under the auspices of a national demon- 
stration project called Opening Doors, Chaffey developed a program designed to increase probation- 
ary students’ chances of succeeding in college. Chaffey’ s program included a “College Success” 
course, taught by a counselor, which provided basic information on study s ki lls and the requirements 
of college. As part of the course, students were expected to complete five visits to “Success Cen- 
ters,” where their assignments, linked to the College Success course, covered skills assessment, 
learning styles, time management, use of resources, and test preparation. 

In 2005, MDRC collaborated with Chaffey College to evaluate the one-semester, voluntary Opening 
Doors program. In 2006, the program was improved to form the two-semester Enhanced Opening 
Doors program, in which probationary students were told that they were required to take the College 
Success course. In MDRC’s evaluation of each program, students were randomly assigned either to 
a program group that had the opportunity to participate in the program or to a control group that 
received the college’s standard courses and services. This report presents the outcomes for both 
groups of students in the Enhanced Opening Doors evaluation for four years after they entered the 
study. The findings include: 

• The message matters — optional program activities had lower participation rates com- 
pared with required program activities. 

• Chaffey’s Enhanced Opening Doors program had positive short-term effects. When the 
two program semesters were complete, students in the program group had earned more credits 
than students in the control group and were nearly twice as likely as control group students to be 
in good academic standing. 

• Despite the program’s encouraging short-term effects, it did not meaningfully improve 
students’ long-term academic outcomes. Eour years after the study began, program and con- 
trol group students had made similar academic progress. Strikingly, during that time, only 7 per- 
cent of all students in the study had earned a degree or certificate. 

This report presents detailed findings from Chaffey’s Enhanced Opening Doors initiative, including 
the cost and cost-effectiveness of the program, and considers the implications of this research for 
designing services for probationary students in community college. 
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Preface 



Like many community colleges, Chaffey College enrolls large numbers of students who 
struggle academically and are eventually placed on probation for their poor academic perfor- 
mance. Many of these students ultimately leave college without earning a credential. “College 
Success” courses, which teach students the skills they need to navigate their way successfully 
through school, are one popular strategy used on numerous college campuses to improve 
students’ chances of succeeding after being placed on academic probation. 

Through a national demonstration project called Opening Doors, Chaffey’ s leaders de- 
veloped a program designed to increase probationary students’ chances of succeeding. The key 
feature of the program is a three-credit College Success course, which is taught by a college 
counselor. As part of the course, students are expected to visit the college’s “Success Centers,” 
where individualized and group instmction in math, reading, and writing is available. 

This report describes Chaffey’s program and its effects on students’ academic outcomes 
four years after students entered the program. This random assignment experimental study 
found that during the two semesters when students were eligible to participate in the program, 
they earned more credits, had a higher grade point average, and were more likely to be in good 
academic standing than students in a control group who did not participate in the program. Most 
of these findings, though, were a result of the credits earned in the College Success course, 
credits that are not applicable toward a degree and cannot be transferred to another institution. 
Nonetheless, the findings are notable because, in spite of adding the College Success class to 
their regular course load, students in the program kept pace with the control group (most of 
whom did not take the course) on academic outcomes. This finding suggested that when the 
program ended, the skills and study habits gained from the course might transfer to other 
classes. However, despite its early promise, the program’s effects were not sustained and did 
not translate into meaningful impacts on students’ academic success over the long term. 

These sobering findings are a reminder of the challenges faced by community colleges 
in general and students on probation in particular. While the Enhanced Opening Doors program 
may be a good start, a more intensive or longer-lasting initiative may be necessary to help 
probationary students beyond the short term. With the long-term results of the Opening Doors 
study at Chaffey now known, administrators, counselors, faculty, and researchers should begin 
to consider what can be done for probationary students in the future. 



Gordon L. Berlin 
President 
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Executive Summary 



Community colleges across the United States face a difficult challenge. On the one hand, they 
are “open access” institutions, with a clear mission to serve students from all backgrounds and 
at varying levels of college readiness. On the other hand, they must uphold high academic 
standards in order to maintain accreditation and adequately prepare students for employment or 
transfer to four-year colleges and universities. How, then, can community colleges best serve 
students who want to learn but do not meet minimum academic standards? 

Chaffey College, a large community college located about 40 miles east of Los Ange- 
les, began to wrestle with this question early in the twenty-first century. At the time, roughly 
one out of every five students enrolled at Chaffey was on probation for poor academic perfor- 
mance (meaning they had a cumulative grade point average, or GPA, below 2.0) or were 
making insufficient progress toward a degree (meaning that they failed to complete half or 
more of the credits they attempted). Under the auspices of a national demonstration project 
called Opening Doors, Chaffey developed a program that was designed to increase probation- 
ary students’ chances of succeeding in college. The program went through two iterations: 
Opening Doors, which began in fall 2005, and Enhanced Opening Doors,' which began the 
following year. The Enhanced Opening Doors program, which is the subject of this report, 
comprised three core components: 

• College Success course. Taught by a college counselor, this “guidance” 
course, which students in the program were told they had to take, was de- 
signed to help probationary students clarify their personal goals, understand 
college rules and regulations, and develop better study skills. A two-credit 
lecture course was linked to a one-credit workshop in which students would 
apply the principles covered in the lecture. The course’s credits counted to- 
ward full-time enrollment in the college and were included in students’ GPA, 
but did not count toward a degree or transfer to a four-year college or univer- 
sity. In addition, a voucher was provided to students in the program to cover 
the cost of College Success course books. A second-semester College Suc- 
cess course, which was not mandatory, was also offered to students in the 
program; however, only a minority of students enrolled in the second- 
semester College Success course. 



'At Chaffey, the second iteration of the program, called “Enhanced Opening Doors” in this report, was 
called “Opening Doors to Excellence.” 
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• Visits to the Success Centers, As part of the College Success course, stu- 
dents were expected to complete five visits to the college’s “Success Cen- 
ters,” where all Chaffey students could get extra help in reading, writing, and 
math. These visits also included assignments for students in the Enhanced 
Opening Doors program, linked to the College Success course, that covered 
the following topics: skills assessment, learning styles, time management, 
use of resources, and test preparation. 

• Improved counseling. The instmctors of the College Success courses were 
expected to work with students during class time and to meet with them out- 
side of class to provide extra advising and counseling as needed. 

In order to determine the effectiveness of Chaffey’ s Enhanced Opening Doors program, 
MDRC — the nonprofit, nonpartisan organization responsible for launching and managing the 
Opening Doors demonstration — conducted a random assignment evaluation. MDRC randomly 
assigned 444 students either to a program group that was eligible to participate in the Enhanced 
Opening Doors program as described above or to a control group whose members could take a 
College Success course (though very few did), could visit the Success Centers on their own, and 
could access the college’s standard counseling services. Random assignment ensures that both 
observed characteristics (for example, race and gender) and unobserved characteristics (for 
example, tenacity and motivation) are distributed similarly between the two research groups. 
Consequently, subsequent differences in academic outcomes — known as impacts — can be 
confidently attributed to the program, rather than to preexisting differences between the two 
research groups. 

Early results from the evaluation found that the program led to positive results during 
the two semesters that students were enrolled in the program (the “program semesters”).^ This 
report extends follow-up on the Chaffey sample to four years after students entered the study to 
determine whether the Enhanced Opening Doors program continued to help students perform 
academically, persist in college, and earn college degrees at Chaffey or at other institutions to 
which they transferred during the study period. 

The key findings from this report are: 

• The message matters — optional program activities had lower partici- 
pation rates compared with required program activities. 



^Susan Scrivener, Colleen Sommo, and Herbert Collado, Getting Back on Track: Effects of a Community 
College Program for Probationary Students (New York: MDRC, 2009). 
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The original program, which is not the main focus of this report, offered students the 
chance to participate in the program’s College Success course, but did not require that they 
participate. In this first iteration of the program, only about half the students who were assigned 
to the program group enrolled in the class, believing that it was optional. 

In the Enhanced Opening Doors program, which began in the 2006-2007 academic 
year, students who were randomly assigned to the program group were informed that participa- 
tion in the first semester of Enhanced Opening Doors was required, and that they would not be 
able to register for other classes unless they signed up for the College Success course. Participa- 
tion in the first-semester College Success course jumped to 72 percent (from 52 percent in the 
original Opening Doors program). Although college administrators ultimately decided not to 
enforce the “requirement” for students who did not comply, evidence suggests that the change 
in message led to a higher rate of compliance, an important lesson from this project that may be 
applicable more broadly. 

• The short-term effects of Chaffey’s Enhanced Opening Doors program 
were encouraging. 

The Enhanced Opening Doors program was a two-semester intervention, as noted earli- 
er. Students in the program group and control group had significantly different experiences 
during the first semester of the program. Most program group students took the College Success 
course, whereas control group students did not, and program group students visited the Success 
Centers more often than did their control group counterparts. In theory, completing the College 
Success course and visiting the Success Centers will enable students to perform better academi- 
cally during the program period and to achieve longer-term success as a result of the skills and 
study habits they gain through these experiences. During the second semester of the program, in 
spring 2007, the experiences of program and control group students once again differed, 
although not nearly as dramatically as they had during the first program semester. 

When the two program semesters were complete, students in the program group had 
earned more credits than students in the control group and were more likely to have had a 
cumulative GPA of over 2.0. (See Table ES.l.) These positive program effects were partly 
driven by the College Success course, whose associated credits cannot be applied toward a 
degree and cannot be transferred to a four-year college or university. When credit accumulation 
is examined including only degree-applicable credits and credits earned through developmental 
education classes (which are not applicable toward a degree but count toward enrollment), and 
the College Success course is excluded, program and control group members were observed to 
have earned similar numbers of credits during the first two semesters of the study. However, 
most program group students took at least one semester of the College Success course during 
the first two program semesters in addition to their regular course load — so the fact that 
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The Opening Doors Demonstration 
Table ES.l 



Academic Outcomes at Chaffey College After Two Enhanced Opening Doors 

Program Semesters 

Chaffey College Four- Year Follow-Up Report 



Outcome 


Program 

Group 


Control 

Group 


Difference 

(Impact) 


Standard 

Error 


Cumulative credits earned 


8.4 


5.7 


2 7 


0.7 


Cumulative GPA higher than 2 . 0 ‘^ (%) 


35.7 


23.6 


22 1 *** 


4.3 


Cumulative degree-applicable GPA higher than 2.0'’ (%) 


30.3 


23.2 


7.2 ^ 


4.2 


Ever in good academic standing"’ (%) 


30.4 


15.9 


24 5 iH** 


4.0 


Sample size (n = 444) 


224 


220 







SOURCES: MDRC calculations from Chaffey College transcript and probation data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by round of random assignment. 

Cumulative measures are based on the two program semesters only. 

“Grades earned in all courses except for developmental courses are used in the calculation of grade point 
average (GPA). Students who do not have a GPA for this time period are included as zeros. 

'’Grades earned only in degree-applicable courses, which exclude the College Success course, are used in the 
calculations of GPA. Students who do not have a GPA for this time period are included as zeros. 

“Sample members who were not enrolled were not considered to be in good academic standing. 



program group students did not fall behind control group members on degree-applicable and 
developmental credits may be viewed as a positive finding. In other words, despite having an 
additional three-credit class as part of their course load, program group students managed to 
keep pace with their control group counterparts on their other credits. 

In addition, the program had a positive impact on GPA for degree-applicable courses 
(that is, excluding the College Success course) during the two program semesters, suggesting 
that the positive program effects were not solely a result of the credits and grades in the College 
Success course. Finally, compared with their control group counterparts, students in the pro- 
gram group were nearly twice as likely to be in good academic standing at the end of the two- 
semester follow-up period. In sum, just after the program services were complete, program 
group students’ academic outcomes looked promising. 
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• Despite the program’s encouraging short-term effects, it did not mean- 
ingfully improve students’ long-term academic outcomes. 

This report presents results up to and including four years after students entered the 
study — an additional three years beyond the earlier short-term follow-up. At that point, it is 
expected that any positive short-term effect of the program would have translated into students 
making more progress toward a degree with respect to degree-applicable credits, which a 
student must earn in order to graduate, as well as developmental credits, which students are 
encouraged to complete prior to enrolling in certain degree-applicable courses. As shown in 
Table ES.2, the evidence from this study does not indicate that the program had significant, 
positive, long-term effects (compared with Chaffey College’s regular services) with respect to 
degree-applicable plus developmental credit accumulation, continued enrollment, or attainment 
of a degree or certificate. In general, students who were offered the opportunity to participate in 
the Enhanced Opening Doors program had academic outcomes that were similar to those of the 
students in the control group during the four-year follow-up period. After four years, program 
group members maintained their edge on total credits earned, but this impact mostly reflects the 
three-credit College Success course that students took during the first semester of the study, 
which does not help them move closer to earning a degree or certificate. Strikingly, four years 
after the study began, only 7 percent of all students in the study had earned a degree or certifi- 
cate, and 44 percent were still enrolled in school (with 21-22 percent still at Chaffey College). 

• The Chaffey College Enhanced Opening Doors program had a net cost 
of around $1,300 per program group member over the course of the two 
program semesters. 

The net cost of $1,300 per program group member represents the difference between 
the gross cost per program group member ($4,300) and the gross cost per control group member 
($3,000). One way to lower the cost of the program and potentially improve its overall cost- 
effectiveness would be to make the Enhanced Opening Doors program a one-semester, mandat- 
ed intervention. 

• When all credits are considered, including the College Success course, 
the cost per credit earned for the program group was slightly lower than 
for the control group. However, the cost-effectiveness of earning credits 
fades away when only degree-applicable and developmental credits are 
considered. 

Specifically, the cost per credit earned for program group members ($516 per credit) 
was 3 percent less expensive than the cost per credit earned for control group members ($530 
per credit). This finding is the result of program group members passing a higher percentage of 
their attempted credits compared with control members. However, when credits associated with 
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The Opening Doors Demonstration 
Table ES.2 



Primary Academic Outcomes, Enhanced Opening Doors, 
Four Years After Random Assignment 

Chaffey College Four-Year Follow-Up Report 



Outcome 


Program 

Group 


Control 

Group 


Difference 

(Impact) 


Standard 

Error 


Number of semesters enrolled at any college 


4.18 


4.06 


0.12 


0.24 


Cumulative degree-applicable and developmental 


credits earned'*’ 


18.4 


17.1 


1.4 


1.9 


Enrolled at any college in year 4 or earned 


a degree or certificate (%) 


45.5 


45.5 


0.1 


4.7 


Enrolled at any college in year 4 


44.2 


44.1 


0.1 


4.7 


Earned a degree or certificate 


7.6 


6.4 


1.2 


2.4 


Sample size (n = 444) 


224 


220 







SOURCES: MDRC calculations from Chaffey College transcript, California Community College Chancellor's 
Office, and National Student Clearinghouse data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by round of random assignment. 

Enrollment and credits earned are based on data from fall 2006 through summer 2010. Degree and certificate 
data are from fall 2006 through spring 2010. 

“Includes credits earned in courses taken during the first four years of the study and excludes credits from the 
College Success course. If a student receives a letter grade in a course, this measure only includes credits in 
which the student earned a “C” or higher. If a student takes a class “pass/fail” or if the class is only offered 
“pass/fail,” then this measure only includes a grade of “pass.” 

'’At any public, two-year California college. 



the College Success course are excluded and only degree-applicable and developmental credits 
are considered, the relative cost-effectiveness of the program fades away. Specifically, the cost 
per degree-applicable and developmental credit earned for the program group ($797 per credit) 
was 26 percent more expensive than the cost per degree-applicable and developmental credit 
earned for the control group ($635 per credit). 



Conclusion 

Chaffey College is one of only a small percentage of community colleges in the United States 
that has been willing to subject one of its programs to a rigorous, random assignment evalua- 



6 








tion. Because of the college’s commitment to serving its students as well as possible, the staff 
members at Chaffey were eager to know how effective the Enhanced Opening Doors program 
was compared with the college’s regular services for probationary students. Their willingness to 
participate in this study should be applauded, as it enables them to better serve their students 
and provides other college administrators, policymakers, and researchers with tmstworthy 
information and evidence on which to base their decisions. Although there is not strong evi- 
dence that the Enhanced Opening Doors program has long-term outcomes that are significantly 
different from those produced by Chaffey’ s usual services, there are lessons that the college and 
policymakers can take away from this study. 

Eor example, the Opening Doors study at Chaffey College began as a study of one 
program (the Opening Doors program), and that program evolved into a stronger second 
program (the Enhanced Opening Doors program). A prior MDRC report describes the differ- 
ences between the two programs and their evolution in detail, but one of the main observations 
in the programs’ development was a change from optional participation in certain program 
services to telling students that they were required to participate, which seemed to have effects 
on program participation and short-term program impacts.^ Rates of participation in the 
College Success course were low in the original program, which led Chaffey administrators to 
require participation in the College Success course in the Enhanced Opening Doors program, 
and they told students that their registration would be blocked if they did not comply. This 
change in policy and messaging was associated with a large increase in program participation. 
It is sometimes the case that those individuals who are at the greatest risk of failure are also 
those who are the least likely to participate in programs that are designed to help them succeed. 
College administrators must weigh their desire to allow their students autonomy and decision- 
making power against the fact that doing so may reduce participation in the very programs that 
have been created for their benefit. Clearly, this is a difficult balance to achieve — requiring 
individuals to participate in a program without their buy-in can backfire, yet allowing complete 
flexibility, especially when dealing with students who have a low likelihood of success, may 
not be in the students’ best interest. 

A final lesson that has been emerging from a number of community college studies is 
that one- and two-semester interventions may not be sufficient to make a lasting difference.'* 
Students often do better while they receive interventions, but the impacts fade once the interven- 

^Susan Scrivener, Colleen Sommo, and Herbert Collado, Getting Back on Track: Effects of a Community 
College Program for Probationary Students (New York; MDRC, 2009). 

^^See, for example, Susan Scrivener and Michael Weiss, More Guidance, Better Results? Three-Year Ef- 
fects of an Enhanced Student Services Program at Two Community Colleges (New York: MDRC, 2009); Evan 
Weissman, Kristin F. Butcher, Emily Schneider, Jedediah Teres, Herbert Collado, and David Greenberg, 
Learning Communities for Students in Developmental Math: Impact Studies at Queensborough and Houston 
Community Colleges (New York: MDRC, 201 1). 
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tions end. While it is not realistic — and probably not advisable — for interventions to last 
indefinitely, program operators might consider whether they can do more to help students make 
a smooth transition to regular college services once an intervention like Enhanced Opening 
Doors ends. For example, program operators might place greater emphasis on mapping out 
what courses students should take once the intervention ends and conducting periodic follow-up 
to make sure that students remain on track and have not encountered new barriers to success. 
Some experts have suggested that community colleges need to institute better policies and 
create clearer pathways that will help all students complete the requirements for earning a 
certificate or degree as quickly as possible.^ Some promising new initiatives, such as the City 
University of New York’s Accelerated Study in Associate Programs (ASAP) and the Bill & 
Melinda Gates Foundation’s Completion by Design Initiative,’ are aspiring to do just that, and 
may soon offer lessons on how it might be achieved. 



^See, for example, Davis Jenkins, “Get with the Program: Accelerating Community College Students’ 
Entry into and Completion of Programs of Study,” CCRC Working Work Paper No. 32 (New York: Commu- 
nity College Research Center, Teacher’s College, Columbia University, 2011); Postsecondary Success Team, 
Completion by Design Concept Paper (Seattle: Bill & Melinda Gates Foundation, 2010). 

^See www.cuny.edu/academics/programs/notable/asap/about.html. 

’See www.completionbydesign.org. 
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2005. Dan Bloom, Colleen Sommo. 

Promoting Student Success in Community College and Beyond 
The Opening Doors Demonstration 

2005. Thomas Brock, Allen LeBlanc, with Casey MacGregor. 

Support Success 

Services That May Help Low-Income Students Succeed in Community College 
2004. Rogeair Purnell, Susan Blank, with Susan Scrivener, Reishma Seupersad. 

Changing Courses 

Instructional Innovations That Help Low-Income Students Succeed in Community College 
2003. Richard Kazis, Marty Liehowitz. 

Money Matters 

How Financial Aid Affects Nontraditional Students in Community Colleges 
2003. Victoria Choitz, Rebecca Widom. 

Supporting CalWORKs Students at California Community Colleges 
An Exploratory Focus Group Study 
2003. Laura Nelson, Rogeair Purnell. 

Opening Doors 

Students ’ Perspectives on Juggling Work, Family, and College 

2002. Lisa Matus-Grossman, Susan Gooden, with Melissa Wavelet, Melisa Diaz, Reishma 
Seupersad. 

Welfare Reform and Community Colleges 
A Policy and Research Context 

2002. Thomas Brock, Lisa Matus-Grossman, Gayle Hamilton. 

Opening Doors to Earning Credentials 

Impressions of Community College Access and Retention from Low-Wage Workers 
2001. Lisa Matus-Grossman, Susan Tinsley Gooden. 

Opening Doors 

Expanding Educational Opportunities for Low-Income Workers 
Published with the National Governors Association Center for Best Practices 
2001. Susan Golonka, Lisa Matus-Grossman. 



NOTE; All MDRC publications are available for free download at www.mdrc.org. 
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About MDRC 



MDRC is a nonprofit, nonpartisan social policy research organization dedicated to learning 
what works to improve the well-being of low-income people. Through its research and the 
active communication of its findings, MDRC seeks to enhance the effectiveness of social and 
education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is best known 
for mounting rigorous, large-scale, real-world tests of new and existing policies and programs. 
Its projects are a mix of demonstrations (field tests of promising new program approaches) and 
evaluations of ongoing government and community initiatives. MDRC’s staff bring an unusual 
combination of research and organizational experience to their work, providing expertise on the 
latest in qualitative and quantitative methods and on program design, development, implementa- 
tion, and management. MDRC seeks to learn not just whether a program is effective but also 
how and why the program’s effects occur. In addition, it tries to place each project’s findings in 
the broader context of related research — in order to build knowledge about what works across 
the social and education policy fields. MDRC’s findings, lessons, and best practices are proac- 
tively shared with a broad audience in the policy and practitioner community as well as with the 
general public and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range of policy 
areas and target populations. Once known primarily for evaluations of state welfare-to-work 
programs, today MDRC is also studying public school reforms, employment programs for ex- 
offenders and people with disabilities, and programs to help low-income students succeed in 
college. MDRC’s projects are organized into five areas: 

• Promoting Family Well-Being and Child Development 

• Improving Public Education 

• Promoting Successful Transitions to Adulthood 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local govern- 
ments, public school systems, community organizations, and numerous private philanthropies. 




